The present invention is related to efficient communications using active messages and is more particularly related to sending active messages from a sender to a receiver in a multinode data processing system.
Remote Memory Copy (RMC) (also known as get/put) and Active Messages (AM) have been proposed as alternatives and/or enhancements to standard send/receive messaging protocols typified by the Message Passing Interface (MPI) standard. Active Message is a one-sided communications protocol which allows the origin task to asynchronously invoke a handler in a target task. This is because the Remote Memory Copy interface is generally more flexible in expressing required communication, and Active Messages allow the messaging Application Programming Interface (API) to be easily extended and customized by users to better fit their specific needs. Further, the user can choose to have communications operate in either polling mode or interrupt mode based on the requirements of the user's application.
But for practical use of these Application Program Interfaces (APIs), they must be defined so that they allow for efficient, performance-optimized implementations. A typical Active Message call specifies the data to be moved to the target process, and the address of a handler to be invoked at the target process to manipulate the arriving data in the desired manner. The handler routine is user written and this provides the mechanism for customizing the API to the application's requirements. The handler is responsible for extracting the data from the network, manipulating it as required, and storing the result in the process's target buffer.
Such a definition implies that the whole message has already been received and is supplied to the handler in its entirety when the handler is called. Since a large message is typically sent as a number of smaller network packets which can arrive out of order, these packets must be first assembled (copied) in some buffer by the communications subsystem (CSS), before the handler can be invoked. This implies an additional copy that can be avoided in some instances if the user can directly supply the target buffer where the incoming data should go. Also, to enhance the concurrency of multiple incoming messages, the CSS would have to do complex buffer management. Such buffer management by the CSS, in the absence of any application-specific knowledge, cannot be implemented in the most efficient manner.
Generally, reliable communication is a requirement; that is, applications must be guaranteed that the messages will be delivered, and/or guaranteed that the application will be notified whenever a message is not delivered. This typically requires that each packet or message be "acknowledged." The target process must send back an "acknowledgment message" to the originating process after the message is successfully received at the target. Similarly, for properly flow-controlled communications, message tokens have to be returned periodically via messages. When working in interrupt mode (where arriving messages cause an interrupt at the target), the overhead of processing these secondary "internal" messages is especially undesirable because typically the interrupt handling overhead is very high.
Key performance metrics for messaging libraries are the communication latency and bandwidth. Reliable and flow-controlled message processing requires various different book-keeping functions to be performed as well as introduces various additional "internal" messages in addition to the data transfer itself. If implemented naively, this can significantly add to latency and degrade bandwidth. The key to improving these metrics is to do as much of the book-keeping functions and processing of the "internal" messages off the critical data-transfer path.
Finally, management of internal storage used for keeping control information is an issue. This storage is always at a premium and proper trade-off is required in order to balance the performance and efficiency with the amount of storage to be set aside for this information.